Skip to content

E2E: per-test store isolation with robust teardown#7279

Draft
phyllis-sy-wu wants to merge 1 commit intopsyw-0413-E2E-temporary-time-limit-bumpfrom
psyw-0413-E2E-store-isolation
Draft

E2E: per-test store isolation with robust teardown#7279
phyllis-sy-wu wants to merge 1 commit intopsyw-0413-E2E-temporary-time-limit-bumpfrom
psyw-0413-E2E-store-isolation

Conversation

@phyllis-sy-wu
Copy link
Copy Markdown
Contributor

@phyllis-sy-wu phyllis-sy-wu commented Apr 13, 2026

WHY are these changes introduced?

E2E tests currently share a single pre-existing dev store configured via E2E_STORE_FQDN. This causes:

  • Shared state: earlier test failures can leave installed apps or modified config that affect later tests
  • No isolation: tests can't run independently — one test's side effects leak into the next
  • Fragile teardown: the old teardownApp silently skips steps, doesn't verify success, and a single failure blocks later cleanup

This PR also lays the foundation for parallel workers (future PR), where shared stores would be impossible since multiple tests run simultaneously.

WHAT is this pull request doing?

Per-test store isolation:

  • Tests that need a dev store (app dev, hot reload, multi-config dev, toml-config dev) create a fresh store via browser automation on admin.shopify.com/store-create/organization/{orgId}
  • Each test gets its own isolated store — no shared state between tests
  • Store is created in test setup and cleaned up in teardown

Robust teardown with escalating retry (setup/teardown.ts):

  • 3 independent phases: uninstall app → delete store → delete app
  • Each phase verifies success (e.g., "Add apps to your store" empty state, "Your plan was canceled" page)
  • Escalating retry: retry same step → go back one step and retry both
  • 90s timeout prevents teardown from blocking the suite
  • Failure summary tells you what wasn't cleaned up and to run pnpm test:e2e-cleanup
  • Phase 3 (delete app) always runs even if store cleanup failed

Tagged logging (setup/env.ts):

  • createLogger(tag) produces [e2e][w0][browser], [e2e][w0][cli] format
  • e2eSection() prints section headers: ----- SETUP: store ... -----, ----- TEARDOWN: app ... -----
  • Worker index in all log lines for future parallel debugging

Server error handling (setup/browser.ts):

  • refreshIfPageError(page) detects 500/502 error pages and auto-refreshes
  • Used in dashboard navigation and app deletion retry loops

Files changed:

File Change
setup/store.ts New: store creation, deletion, uninstall, storeTestFixture, isAppsPageEmpty()
setup/teardown.ts New: teardownAll() with escalating retry + 90s timeout
setup/app.ts Extract findAppOnDevDashboard() (with pagination), deleteAppFromDevDashboard(), remove old teardownApp
setup/env.ts Add createLogger(), e2eSection(), workerIndex
setup/cli.ts Worker-tagged logging for exec/spawn
setup/browser.ts Add refreshIfPageError()
All test files Switch to teardownAll, store-dependent tests use storeTestFixture

How to test your changes?

# Run all tests with debug output
DEBUG=1 pnpm --filter e2e exec playwright test

# Run headed to watch store creation/teardown
E2E_HEADED=1 DEBUG=1 pnpm --filter e2e exec playwright test

# Skip teardown for debugging (clean up manually after)
E2E_SKIP_CLEANUP=1 pnpm --filter e2e exec playwright test

# Verify no leftovers after a run
npx tsx packages/e2e/scripts/cleanup-stores.ts --list
npx tsx packages/e2e/scripts/cleanup-apps.ts --list

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing, so I've added a changelog entry with pnpm changeset add

Copy link
Copy Markdown
Contributor Author

phyllis-sy-wu commented Apr 13, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@phyllis-sy-wu phyllis-sy-wu changed the base branch from psyw-0413-E2E-one-time-auth to graphite-base/7279 April 13, 2026 23:38
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-store-isolation branch from a47982f to d6a4e19 Compare April 13, 2026 23:38
@phyllis-sy-wu phyllis-sy-wu changed the base branch from graphite-base/7279 to psyw-0413-E2E-improve-error-context-in-assertions April 13, 2026 23:38
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-store-isolation branch from d6a4e19 to 9659cf3 Compare April 14, 2026 00:01
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-improve-error-context-in-assertions branch from 94f4908 to d872975 Compare April 14, 2026 00:01
@phyllis-sy-wu phyllis-sy-wu changed the base branch from psyw-0413-E2E-improve-error-context-in-assertions to graphite-base/7279 April 14, 2026 02:45
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-store-isolation branch from 9659cf3 to 2bb6ce4 Compare April 14, 2026 02:45
@phyllis-sy-wu phyllis-sy-wu changed the base branch from graphite-base/7279 to psyw-0413-E2E-temporary-time-limit-bump April 14, 2026 02:45
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-store-isolation branch from 2bb6ce4 to e5289d3 Compare April 14, 2026 03:34
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-store-isolation branch from e5289d3 to 5c0d260 Compare April 14, 2026 04:50
@phyllis-sy-wu phyllis-sy-wu force-pushed the psyw-0413-E2E-temporary-time-limit-bump branch from 77e5ea0 to 5765d59 Compare April 14, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant